Skip to content

feat(cala): Phase 6 — archive + vitals UI#140

Merged
daharoni merged 18 commits into
mainfrom
feat/cala-phase-6
Apr 19, 2026
Merged

feat(cala): Phase 6 — archive + vitals UI#140
daharoni merged 18 commits into
mainfrom
feat/cala-phase-6

Conversation

@daharoni
Copy link
Copy Markdown
Contributor

@daharoni daharoni commented Apr 19, 2026

Summary

Phase 6 (design §12) ships the "feels alive" layer on top of the Phase 5 streaming pipeline:

  • W4 archive backend: tiered timeseries store (L1 full-res + L2 block-averaged), per-neuron event index, log-spaced footprint history snapshots
  • W2 emissions: vitals metrics (cell count, fps, memory, residual L2, extend queue depth), periodic footprint snapshots at ages 1, 2, 4, 8, …, and real extend cycles running inside the fit worker via new `Extender` WASM bindings
  • Dashboard UI: header vitals bar with canvas sparklines, scrolling event feed, "Deprecate latest" user-authored mutation button
  • Deploy: `coi-serviceworker` for GitHub Pages cross-origin isolation so `SharedArrayBuffer` boots on the static host

Exit proven by `apps/cala/e2e/phase6-exit.e2e.test.ts` — full end-to-end run asserting tiered timeseries, per-neuron event queries, footprint history, archive dump, and user mutation round-trip.

Deferred to Phase 7+ (documented in design §12): real `birth`/`merge` `PipelineEvent`s (needs `Fitter.drainApplyEvents` binding), cross-worker snapshot transport so extend truly runs in W3, ε change-triggered footprint snapshots, Playwright harness, click-a-footprint mutation UX.

Test plan

  • `cargo test -p calab-cala-core` passes (extend driver extracted, E2E parity preserved)
  • `bun test` passes across `@calab/cala-runtime` + `apps/cala` (new unit tests for timeseries store, event index, footprint history, snapshot scheduler, vitals store, archive client, event-format)
  • `apps/cala/e2e/phase6-exit.e2e.test.ts` green
  • `apps/cala/e2e/phase6-extend.e2e.test.ts` green
  • `apps/cala/e2e/phase5-exit.e2e.test.ts` still green (StubExtender mock added after task 11 made Extender import unconditional)
  • Live browser smoke: vitals bar populates, event feed scrolls, "Deprecate latest" emits a deprecate event and cell count drops

🤖 Generated with Claude Code

daharoni and others added 18 commits April 18, 2026 21:57
Upgrade the archive worker from a latest-value Map to per-name
tiered Float32Array rings (design §9.1): L1 full-resolution recent +
L2 block-averaged older. Adds `request-timeseries` / `timeseries`
protocol variants so the dashboard can pull either tier. Existing
`archive-dump` shape (events + latest-value metrics) is unchanged,
keeping task 24's dashboard working without edits.

Capacities and stride are all overridable via `workerConfig`, no
magic numbers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `NeuronEventIndex` — bounded drop-oldest per-neuron ring of
every structural event the neuron participates in (design §9.2), so
"show me the history of neuron 47" is O(index) instead of a scan of
the global event ring. Wires a third bus subscriber in the archive
worker plus a `request-events-for-neuron` / `events-for-neuron`
protocol pair.

Also bumps the archive worker's local-bus `maxSubscribers` default
to 8 so the upcoming footprint-history subscriber fits without
another edit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the archive-side receiver for the hybrid log-spaced +
change-triggered footprint scheme (design §9.3). The new
`FootprintHistoryStore` keeps per-neuron drop-oldest rings of
`(t, sparse A column)` snapshots; typed-array payloads are retained
by reference to avoid per-snapshot copies.

Harvests footprints from three sources through the archive worker's
local event bus: (1) birth events, (2) merge-survivor events, (3)
split children, plus a new `footprint-snapshot` `PipelineEvent`
variant that W2 will emit on the log-spaced schedule in task 5.
Adds a `request-footprint-history` / `footprint-history` protocol
pair for the scrubber UI in Phase 7. Default sizing matches the
§9.3 ~5 MB per-session budget.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fit worker now emits the five header-bar vitals (cell_count, fps,
memory_bytes, residual_l2, extend_queue_depth) every vitalsStride
frames (default 8). residual_l2 is computed from `fitter.step()`'s
return; fps is wall-clock-derived over the last interval so it
reflects what the user sees on the sparklines. Metric names live in
a new `lib/vitals.ts` module shared with the upcoming vitals bar so
emitter and UI can't drift.

Also adds `calaMemoryBytes()` to the cala-core wasm-adapter — a
single source of truth for the WASM heap size now that a consumer
(W2) actually needs it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the log-spaced floor of the §9.3 hybrid scheme: a new
`FootprintSnapshotScheduler` emits `footprint-snapshot` events at
ages 1, 2, 4, 8, … frames after each neuron's birth, using the
latest footprint attached to a mutation (register / merge / split)
as its cached snap.

The change-triggered branch (‖A_curr − A_last_snap‖_F drift) is
left as a deliberate TODO — it needs a new wasm-bindgen accessor on
`Fitter` to read per-component A columns. Until that lands the
log-spaced schedule gets us scrubber data during quiet periods and
exercises the archive's footprint-history store end-to-end.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the main-thread archive client with three new promise-based
request methods that forward to the W4 queries added in tasks 1-3:

  - requestTimeseries(name)     → tiered sparkline data for a metric
  - requestEventsForNeuron(id)  → per-neuron structural history
  - requestFootprintHistory(id) → per-neuron (t, sparse A) scrubber data

Refactors the existing dump path onto a generic `issueRequest`
helper so every reply kind shares one requestId-correlation store
without type drift.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the §12 header vitals bar: five Canvas-based sparkline widgets
bound to a new `vitals-store` that polls the archive client every
500 ms for cell_count / fps / memory_bytes / residual_l2 /
extend_queue_depth. L1 + L2 tiers merge into a single flat window
(default 120 samples ≈ 60 s of history) so the component doesn't
know about the retention scheme.

Lifecycle is tied to the run: the bar spins up an archive client
on transition to running, polls while the run is active, and tears
down on stop/error. Sparklines auto-scale per series so each vital
gets its full vertical range regardless of magnitude.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the §12 event feed: a structured, grid-laid-out log of every
pipeline event from the dashboard store — kind, id, time, and a
compact human-readable description per row. Birth / deprecate /
reject rows get distinct accent colors so the feed reads as a
narrative rather than a wall of text.

Format helpers are extracted into `event-format.ts` and
unit-tested directly so the visual component stays untouched by
string-formatting churn. Task 9 composes this + VitalsBar +
SingleFrameViewer into the final dashboard layout.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Composes VitalsBar + SingleFrameViewer + EventFeed into a single
grid layout (design §12): vitals along the top, preview canvas on
the left, event feed on the right, each cell independently
scrollable. App.tsx hands off to `DashboardLayout` on any
active-run state; the old SingleFrameViewer side-panel + inline
event list come out since their roles are now covered by the
dedicated components.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extracts the Phase 3 cold-start E2E's inline `run_extend_cycle`
helper into a public crate function at
`extending::driver::run_cycle`, then wraps it behind a new
`Extender` wasm-bindgen class. The browser W3 worker can now own a
`ResidualRingBuf`, push residuals per fit frame, and call
`runCycle(fitter, queue)` on whatever cadence it chooses —
proposals land on the shared `MutationQueueHandle` ready for
`drainApply`.

Exports `Extender` through `@calab/cala-core`. The Phase 3 test
switches to the shared driver to prove numerical parity: same code
path serves both the native cold-start recovery proof and the
browser worker.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fit worker now owns an `Extender` + residual ring: on every frame
it pushes the residual, and on `extendCycleStride` (default 32
frames ≈ 1 s at 30 fps) it calls `Extender.runCycle(fitter, queue)`.
Real `register` proposals land on the Rust mutation queue and get
applied on the next `drainApply`, advancing the fitter's epoch and
populating `cell_count` / `numComponents` — the vitals bar actually
moves now.

The per-cycle proposal count is published as `extend.proposed` so
the event feed shows extend activity even before structural events
surface. W3's heartbeat stub stays in place for the orchestrator
lifecycle handshake.

Architectural trade-off: running the cycle inside W2 avoids a
cross-worker snapshot transport (design §7.2). True W2/W3
separation needs a WASM binding that serializes the Snapshot across
workers + a `Fitter.drainApplyEvents()` to surface `register`
payloads as `birth` events — both deferred to Phase 7.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `e2e/phase6-extend.e2e.test.ts`, mirroring the Phase 5 exit
harness (real AVI bytes, real SabRingChannel, real worker modules)
but configured with a non-zero `extendCycleStride` so fit's new
extend path actually fires. Asserts:

  * Extender.runCycle is driven every stride
  * fitter.drainApply is called per cycle → epoch advances
  * `extend.proposed` metric events reach W4 with the expected count

Also fixes a bug the E2E surfaced: task 11's `runExtendCycleIfDue`
pushed proposals onto the Rust mutation queue but never triggered
`drainApply` — the existing drain loop only fires when the JS-side
queue has items. Now the extend path drain-applies inline so epoch
advances as soon as the cycle proposes anything.

Phase 5 E2E mock picks up a no-op StubExtender since the fit
worker's Extender import became unconditional; no behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a main-thread authoring path for pipeline mutations. New
types and flow:

  * `UserMutation` in the runtime's worker-protocol (narrow to
    `deprecate` for Phase 6 — register / merge need a footprint
    picker, deferred).
  * `RuntimeController.pushUserMutation(m)` forwards to the fit
    worker as a `user-mutation` inbound.
  * Fit worker handles it by pushing through the Rust-side queue
    (`pushDeprecate`), drain-applying so epoch advances, and
    publishing a `deprecate` structural event so the UI feed shows
    the user's action.

UI affordance: a "Deprecate latest" button in the event feed
toolbar that targets the most recently born neuron id. The real
click-a-footprint UX lands with the Phase 7 overlay.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops `public/coi-serviceworker.js` — an inlined, BSD-licensed
service worker that re-issues top-level navigations with COOP +
COEP headers so the Pages deploy boots `crossOriginIsolated` and
`SharedArrayBuffer` works without server-side header control.
Registered synchronously in `index.html` before the module script
so the page is isolated before any SAB-using worker is constructed.

Vite dev + preview already set the same headers directly, so this
is a no-op there.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…sk 16)

Ships `e2e/phase6-exit.e2e.test.ts`: drives the full decode → fit
→ archive pipeline on a real miniscope AVI and asserts every
Phase-6-shipped surface is reachable end-to-end:

  - vitals timeseries populate (tiered store from task 1)
  - per-neuron event index resolves a birth (task 2)
  - footprint history returns entries (task 3)
  - extend.proposed metrics + structural events land in the archive
    dump (tasks 4 + 5 + 11)
  - user-authored deprecate mutations reach the fit worker (task 13)
  - no worker errors during or after the run

Also fixes `pixel_size_um` missing from W2's metadataJson (task 11
introduced an Extender construction that parses RecordingMetadata
but run-control.ts only forwarded height/width). Surfaced during
live browser testing; without the fix the fit worker panics on
init in production.

Updates `.planning/CALA_DESIGN.md` §12 with the Phase 6 exit date,
the shipped artifact list, and explicitly deferred items (real
birth events need `Fitter.drainApplyEvents`, cross-worker
snapshot transport, ε change-trigger, Playwright, click-a-
footprint UX).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
W1's `frame-processed` heartbeat hardcodes `epoch: 0n` (it has no
view of the fit pipeline's epoch), but `run-control` was routing
that to `recordFrameProcessed` — so the viewer label read
`epoch 0` forever while the fit pipeline silently advanced. Move
the dashboard listener onto the fit worker, which is the only
worker that knows the real epoch. W1's listener stays for
`frame-preview` only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `initCalaCore` resolver now reads `mod.memory` to expose
`calaMemoryBytes()` (Phase 6 task 23 / vitals bar), and the
adapter re-exports the new `Extender` binding (task 23). The
mocked init resolver needed to return a stub `WebAssembly.Memory`
instead of `undefined`, and the stub module needed to export an
`Extender` class. Adds one test for the new `calaMemoryBytes()`
helper.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@daharoni daharoni merged commit 0a44086 into main Apr 19, 2026
7 checks passed
@daharoni daharoni deleted the feat/cala-phase-6 branch April 19, 2026 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant